A Model Curriculum for an Emergency Medicine Residency Rotation in Clinical Informatics

Audience This curriculum is designed for emergency medicine residents at all levels of training. The curriculum covers basic foundations in clinical informatics for improving patient care and outcomes, utilizing data, and leading improvements in emergency medicine. Length of Curriculum The curriculum is designed for a four-week rotation. Introduction The American College of Graduate Medical Education (ACGME) mandated that all Emergency Medicine (EM) residents receive specific training in the use of information technology.1,2 To our knowledge, a clinical informatics curriculum for EM residents does not exist. We propose the following standardized and reproducible educational curriculum for EM residents. Educational Goals The aim of this curriculum is to teach informatics skills to emergency physicians to improve patient care and outcomes, utilize data, and develop projects to lead change.3 These goals will be achieved by providing a foundational informatics elective for EM residents that follows the delineation of practice for Clinical Informatics outlined by the American Medical Informatics Association (AMIA) and the American Board of Preventive Medicine (ABPM).4–6 Educational Methods The educational strategies used in this curriculum include asynchronous learning via books, papers, videos, and websites. Residents attend administrative sessions (meetings), develop a project proposal, and participate in small group discussions. The rotation emphasizes the basic concepts surrounding clinical informatics with an emphasis on improving care delivery and outcomes, information systems, data governance and analytics, as well as leadership and professionalism. The course focuses on the practical application of these concepts, including implementation, clinical decision support, workflow analysis, privacy and security, information technology across the patient care continuum, health information exchange, data analytics, and leading change through stakeholder engagement. Research Methods An initial version of the curriculum was introduced to two separate institutions and was completed by three rotating resident physicians and one rotating resident pharmacist. A brief course evaluation as well as qualitative feedback was solicited from elective participants by the course director, via email following the completion of the course, regarding the effectiveness of the course content. Learner feedback was used to influence the development of this complete curriculum. Results The curriculum was graded by learners on a 5-point Likert scale (1=strongly disagree, 5 = strongly agree). The mean response to, “This course was a valuable use of my elective time,” was 5 (sd=0). The mean response to, “I achieved the learning objectives,” and “This rotation helped me understand Clinical Informatics,” were both 4.75 (sd=0.5). Discussion Overall, participants reported that the content was effective for achieving the learning objectives. During initial implementation, we found that the preliminary asynchronous learning component worked less effectively than we anticipated due to a lower volume of content. In response to this, as well as resident feedback, we added significantly more educational content. In conclusion, this model curriculum provides a structured process for an informatics rotation for the emergency medicine resident that utilizes the core competencies established by the governing bodies of the clinical informatics specialty and ACGME. Topics Clinical informatics key concepts, including definitions, fundamental terminology, history, policy and regulations, ethical considerations, clinical decision support, health information systems, data governance and analytics, process improvement, stakeholder engagement and change management.

core concepts, methods, and skills of clinical informatics. As residents enter an increasingly digital medical world, education must adapt to equip students with the necessary tools to further advancements in clinical care and health systems. It has been suggested that informatics should be a part of all formal medical education. 9 While many medical schools and residencies have added educational content related to information retrieval and basic use of electronic health records, only a few have expanded their curricula to include the myriad of other ways physicians interact with electronic health information including clinical decision support, quality measurement and improvement, personal health records, telemedicine, and personalized medicine, despite established undergraduate medicine competencies and curricula. 10,11 The use of health information technology permeates graduate medical practice, yet only a handful of residency programs and national organizations have designed, integrated, or recommended formalized informatics curriculum into their training programs. [12][13][14][15][16][17] Informatics is especially relevant to the specialty and practice of Emergency Medicine (EM). The efficient management of information from the rapid processing, analysis, and interpretation of patient data in an emergent setting is crucial to patient outcomes. Through the widespread use of electronic health records feeding data to community resources, emergency physicians can monitor and better care for specific populations of patients, thus bridging the gap between personal and population health. The use of technology has become a requirement in EM training from both the American Board of Emergency Medicine (ABEM) Knowledge, Skills and Abilities (KSAs) 18 and the Emergency Medicine Milestones Project, a joint venture between ABEM and the Accreditation Council for Graduate Medical Education (ACGME). 2,19 Specifically, milestone Systems-Based Practice (SBP) 3 focuses on use of technology, while milestone SBP1 focuses on patient safety. Milestone SBP2 focuses on system-based management. Milestone Practice-Based Learning and Improvement (PBLI) focuses on the practice of evidence-based medicine, while Milestone Interpersonal and Communication Skills (ICS) 2 focuses on team management. 2,19 Developing and updating resident competencies in informatics is essential for the future of EM as a specialty, yet informatics education in EM graduate training is often absent or inconsistent in the US and abroad. 20,21 Currently, few options exist for residents to gain exposure to this field. Residents in departments that also host ACGME accredited fellowships may receive more exposure, but only four of these programs were documented in 2018 by the Emergency Medicine Residents Association (EMRA), 22 though other clinical informatics programs offer EM tracks. 23 There are also distance learning opportunities, including AMIA's 10x10 course created by Oregon Health and Science University. One section of this course is held in conjunction with the American College of Emergency Physicians (ACEP) annually. However, these courses span many months and cost approximately $2,000, which may limit the number of EM residents who take the course. 24 A residency curriculum in clinical informatics could help ensure the development of EM physicians who are able to use the tools of informatics to improve care for patients.

Curriculum Development Framework
Using Kern's six-step approach to medical education curriculum development as a framework, 25 we developed a curriculum for EM clinical informatics education that aids the next generation of emergency physicians in utilizing clinical informatics during a graduate medical education elective. We included a thematic breakdown of vital topics in EM informatics, as well as suggested content and readings.

Problem identification, general and targeted needs assessment:
The American College of Graduate Medical Education (ACGME) mandates that all EM residents receive specific training in the use of information technology. 2,19 EM physicians routinely interact with clinical systems in their medical practice. The ABEM and the ACGME mandated that all EM residents receive specific training in the use of information technology to optimize learning, improve patient care, and accomplish and document safe health care delivery. 2,18 Currently, a widely accessible, standardized informatics curriculum is not available to train EM residents. As a result, exposure to foundational clinical informatics remains inconsistent between training programs, leaving many EM residents with a deficit in training. We propose the following standardized and reproducible educational curriculum for EM residents.

Goals of the curriculum:
The aim of this curriculum is to teach informatics skills to emergency physicians to improve patient care and outcomes, utilize data, and develop leadership abilities. These goals will be achieved by providing a foundational informatics elective for EM residents that follows the delineation of practice for clinical informatics outlined by the American Medical Informatics Association (AMIA) and the American Board of Preventive Medicine (ABPM). 5,6 Objectives of the curriculum: Residents will gain an introduction of the broad field of clinical informatics, with a focus on the key applications of informatics in EM. By the end of this rotation, the learner will be able to: 1. State the value proposition of clinical informatics.

Educational Strategies:
The curriculum will offer an introductory study of the basic concepts surrounding clinical information systems, focusing on the practical application of these concepts within the realm of EM. Asynchronous coursework will be combined with experiential learning to create a robust educational experience that can be replicated and implemented at a variety of EM residency training programs.
The curriculum fulfills the ACGME requirements pertaining to information technology for EM programs. Informatics is a rapidly developing field. We suggest the rotation director be an EM physician with experience in the field of clinical informatics, preferably board eligible or board certified. While the educational system is growing with certifications and fellowships, we do not feel these are necessary for the EM residency informatics rotation instructor. Many experts in the field acquired knowledge through experience and self-education. Most of the rotation will take place in the hospital setting. Learners should have access to the institutional health system with basic literacy in information retrieval.
The goal of this curriculum is for residents in EM to build basic competency in the field of informatics through education and clinical application. They will expand their knowledge beyond data access and retrieval. They should understand the broader role of health information technology (HIT) in health care delivery, public health and clinical research, and quality improvement. Clinical informaticians draw from the field of biomedicine and informatics to apply these tools to the practice of medicine. As per the AMIA white paper on Core Content for the Subspecialty of Clinical Informatics, clinical informatics encompasses three spheres of activity: clinical care, the health system, and information and communications technology. Through this elective, residents will gain a better understanding of the five major categories that form the core content of informatics: fundamentals, improving care delivery and outcomes, health information systems, data governance and data analytics, and leadership and professionalism. [4][5][6] I. Fundamentals The first core concept focuses on learning the fundamentals and basic knowledge needed for students to understand the environment in which an informaticist functions. This includes the history of informatics, key informatics concepts, models and theories, commonly cited literature, and an overview of the Residents will acquire knowledge of the fundamentals of informatics through online resources, reading materials, participating in both department and hospital level operations and committee meetings as well as small group sessions.
Additionally, residents will be encouraged to set up one-on-one meetings with university librarians to establish competency with medical literature database searches and information retrieval. They will also meet with Electronic Health Record (EHR) informatics analysts and/or clinical informaticists to learn the basics of their learning environment's EHR, hospital information systems and leadership organization structure. After residents have a foundational knowledge of the key concepts of clinical informatics, they will then have an opportunity to apply these ideas to their clinical workflows. The fundamentals of clinical informatics align with milestones SBP3. 2,19 II. Improving Care Delivery and Outcomes Implementing and integrating clinical decision-making and care process improvement is the second core concept. One of the goals of this elective is for EM residents to understand their own decision-making through effective information analysis. Important concepts include evidence-based medicine (EBM), clinical decision support (CDS), and process redesign. 4 While most students have a rudimentary understanding of clinical decision support and electronic health record capabilities, through independent readings and project assignments, they will understand how to optimize the system capabilities and become better advocates for patient care.
To accomplish this, they will review active and passive CDS in use in the ED, from order sets to interactive alerts, and gain an appreciation for the CDS maintenance processes. Furthermore, they will expand these skills to answer questions regarding individual, institutional, and community health practices. Residents will have the opportunity to apply informatics core concepts to identifiable gaps in the clinical setting and apply process redesign to address those problems. Improving Care Delivery and Outcomes aligns with milestones SBP1, SBP2, SBP3, PBL1 and ICS2. 2,19 III. Health Information Systems The third core concept focuses on health information systems and interoperability. Themes in this concept include computer programming, networks and databases, data security, clinical data standards and information system lifecycles. 4 Information systems are complex networks encompassing people, processes, and technology. As residents move into new roles beyond training, they must not only know how to use health systems, but also how to evaluate the effectiveness of a system in meeting clinical goals. They need to be able to critically assess the advantages and disadvantages of technological tools and interpret the quality of data produced. Residents should not be expected to learn computer programming but instead appreciate the advantages and limitations of information systems and their associated software.
They will accomplish these objectives through independent readings and online tutorials.
The final project in our curriculum will allow residents to develop an informatics-based solution to a problem in their clinical environment. They will explain how they would implement an optimization or introduce a new workflow to their EHR, and develop measures of the effectiveness of that change, in terms of clinician behavioral change or patient outcomes. These skills enable residents to evaluate the reliability of data, provide feedback, and integrate their solution to provide better patient care.

IV. Data Governance and Data Analytics
The fourth concept focuses on standards of data governance, policies, and processes. 5,6 Through readings and online didactics, residents will learn best practices for data use, privacy and security, as well as the "data life cycle" and need for data validation and management to optimize data sharing across systems.
EM residents will gain an understanding of the basic principles of analytics techniques, as well as machine learning, data visualization and natural language processing through independent reading and small group discussion. Residents will apply this knowledge in their final project, maintaining appropriate methodologies of data governance and utilizing their understanding of analytics to propose methods of assessment and interpretation of their described intervention. Finally, the fifth core concept focuses on the organization of health care institutions and techniques to effectively introduce change into these organizations. Themes include identifying and engaging with stakeholders, building interdisciplinary teams, and project management. 4 As medicine becomes increasingly interdisciplinary, physicians must foster collaboration and effect change at the organizational level. It is useful to understand how informatics decisions for the ED can impact a diverse array of stakeholders -from patients and front-line staff to social work and case management, consultants, inpatient and outpatient clinicians, as well as regulatory and compliance, legal and financial stakeholders.

USER GUIDE
While a single rotation will not allow for full development of leadership skills, EM residents will experience some aspects of managing teams, effective communication, and group management processes. It is expected that residents will observe and understand administrative leadership roles by attending meetings with departmental and hospital leadership. Examples of administrative leadership meetings include the following: clinical decision support committee, computerized provider order management committee, quality and utilization committee, medical informatics committee, physician advisory council, analytics council, process excellence committee, and clinical pathways committee.
Residents will gain a basic understanding of leadership and professionalism through the lens of clinical informatics through reading and online video instruction. The residents will apply the knowledge they gained of the five core competencies to a project with an informatics-based solution. This will be based on a problem in their clinical setting. Ideas may include clinical decision support, hospital information systems, data analytics, and leadership skills to enact change.
Through this introduction to clinical informatics, we hope residents will gain a better understanding of the framework of information systems, including processes to enhance medical decision making and improve patient care. Demand will increase in the EM community to invest in informatics initiatives. It is our obligation to train students to optimize available tools and develop new solutions to improve the future of medicine.
Curriculum Design This curriculum aims to provide a model for a structured informatics rotation for EM residents that fulfills ACGME requirements by following the Core Content for Clinical Informatics outlined by the American Medical Informatics Association (AMIA) and the American Board of Preventive Medicine (ABPM). [4][5][6] The authors performed a PubMed (National Center for Biotechnology Information, Bethesda, MD) and Google Scholar search for terms associated with EM residency informatics curricula. We were unable to identify any published EM residency curricula in clinical informatics. Informatics curricula from other non-EM specialties, non-Graduate Medical Education (GME) courses, and medical student electives were adapted to meet the needs of an EM resident. 10,11,[15][16][17]20,[22][23][24] Expert opinion on the fields of EM informatics and education were included for the curriculum design: authors Carrie Baker, Benjamin Slovis, Nicholas Genes and Jeffrey Nielson all hold leadership positions in ACEP with extensive experience in EM informatics education. Benjamin Schnapp is a medical education fellowship trained EM physician with experience in resident education and curriculum design. William Hersh and Vishnu Mohan are leaders in clinical informatics with numerous academic and educational contributions to the field.

Results and tips for successful implementation:
Four learners participated in an initial version of the curriculum and provided feedback via the standardized follow-up survey as well as comments on the curriculum. These results were used to further develop the curriculum into its current state. Surveys included a 5-point Likert scale with 1 indicating "strongly disagree" and 5 indicating "strongly agree." The mean response ,"This course was a valuable use of my elective time," was 5 (sd=0). The mean response to, "I achieved the learning objectives," and "This rotation helped me understand Clinical Informatics," were 4.75 (sd=0.5).
Free-form feedback from participants was also received. Comments included: "I got a good experience out of this rotation. I was able to get a good feeling for what an informatics fellowship could be like." Rotators found that the leadership component was particularly effective, reporting "The exposure to the daily workload and workflow of the fellows greatly helped me to understand and appreciate the role of clinical informatics. Attending various meetings with the network's informatics team allowed me to see the power of teamwork and quality improvement projects." Another learner commented, "It showed me the general mindset and process of thinking as a clinical informaticist in how data is collected, processed, and applied in the clinical setting to take data and create knowledge. I also learned some of the terminology and language used in informatics which is important in understanding and communicating in the field." Targeted learners are either second or third-year EM residents who are expected to be in good standing with the residency program. Residents will have approximately four weeks to complete the curriculum, with asynchronous learning Learners will appreciate the role of health information exchanges in the emergency department.
Learners will appreciate the concept of terminologies and their application in health information technology.
Learners will understand the need for Healthcare Data standards.
Learners will appreciate the concept of data analytics and its application in healthcare.
Learners will develop a basic understanding of the concepts of data analytics for ED needs, including techniques of "AI" / machine learning, and natural language processing.

Objectives:
Residents will gain an introduction of the broad field of clinical informatics, with a focus on the key applications of informatics in EM. By the end of this small group section, the learner will be able to: 1. State the value proposition of clinical informatics. 2. Describe the federal policies and legislation that influence the adoption of health information technology in the United States.

Linked objectives and methods:
Objectives are achieved through small group discussion with guidance from the small group instructor. This allows for knowledge translation in an informal setting. Learners discuss their experiences and ideas in an open format. a. An appropriate definition should include the themes of acquisition, storage, and use of information. While there is a current association with information technology this is not a critical part of the definition itself. b. Informatics is an association between informational science and a domain. It is not doctors "tinkering" with computers, nor large data analysis (though this can be part of the development of informatics interventions). c. It is not health information management, nor solely implementation science. 2. Using your own words, generate an example of Friedman's goal of informatics. Can you think of an example of how the rule can be applied in your clinical workflows? a. Friedman's "fundamental theorem" is the concept that a human plus an information system will perform superiorly to a human alone. In the context of clinical informatics, we generally assume this to be that despite some of their pitfalls, humans working with an associated clinical information system will be superior to one working without an information system. An important note is that the human element is critical to the interaction. 3. Take an example from your clinical work and demonstrate how data can progress to information, then knowledge, then wisdom.
a. An example might be how the data point for a blood test can be used to generate information about the result if there is a trend (ie, serial elevated blood glucose levels). This information could result in knowledge via the diagnoses of a patient's condition (ie, a diagnosis of diabetes). The knowledge of the condition could contribute to wisdom and the appropriate interventions to manage the condition. act. The goal was to stimulate the economy and increase adoption of electronic health records (EHR) with improved quality and safety. c. Over $30B in incentives for the "meaningful use" of EHRs along with workforce development and research. 5. Define interoperability. Describe some of the past issues with it and how some policies are being developed to improve it. Describe an experience from your clinical work that demonstrated use of or a lack of interoperability. a. Interoperability implies the ability for systems to share information, but an important distinction in clinical informatics is not just the ability to share but also make use of the information. b. The HITECH act didn't adequately promote interoperability due to a lack of development of standards, thus leading to a world of EHRs that didn't talk to each other. c. The 21 st century Cures Act attempts to correct some of the issues with interoperability by prohibiting information blocking and setting a standard for information transfer (SMART on FHIR).

Objectives:
Residents will gain an introduction of the broad field of clinical informatics, with a focus on the key applications of informatics in emergency medicine. By the end of this rotation, the learner will be able to: 1. Understand the need for standards, clinical terminologies, and ontologies. 2. Recognize the need for data governance and analytics.

Linked objectives and methods:
Objectives are achieved through small group discussion with guidance from the small group instructor. This allows for knowledge translation in an informal setting. Learners can discuss their experiences and ideas in an open format. This presentation covers three objectives. Grounded in examples and discussion using the documents, "Appendix E.3.b. Data Analytics Governance Instructor Material," and "Appendix E.3.c. Data Analytics Governance Learner Material," learners will: 1. Develop a basic understanding of the concepts of data analytics for ED needs, including techniques of "AI"/machine learning and natural language processing. 2. Understand strategies for data warehouse access and methods of conducting research and quality projects to improve ED care and operations. 3. Appreciate the utility of health information exchange in the ED, as well as some of the challenges of interoperability, necessity of data standards, and tradeoffs involved with different consent models.
As described above, the instructor should be familiar with the cited studies and aspects of running queries and sharing data to facilitate discussion. The documents, "Appendix E.3.b. Data Analytics Governance Instructor Material," and "Appendix E.3.c. Data Analytics Governance Learner Material," serve as a discussion guide for the small group. Method #2 is also relatively easy to build, but requires more work of the callback center. Method #3 is the most labor-intensive, relying on both human curation of the list and interruptive notifications to the callback center.
1Q4: What challenges do you think the callback center will have, in contacting these patients and preventing unscheduled 72-hour returns? 1A4: Patients at higher risk for return visits may be the hardest to reach (undomiciled or inconsistent address, no mobile phone). Also, patients today may prefer messaging -either SMS or through the EHR patient portal -to phone calls.
2. A few residents are interested in a research project on the use of pain medications in the ED. The hypothesis is that decreasing the default adult dose of ibuprofen from 600mg to 400mg across all order sets and preference lists in the ED will lead to greater use of the safer adult dosage without a significant change in the delta for pain scores. They'd like to be able to log into the identified hospital data warehouse self-service query tool to look at historical pain score trends across ED visits for all chief complaints and prospectively assess the impact of the intervention.
2Q1: What retrospective data may be possible to analyze first, to help answer the study question, before any prospective changes are made to ibuprofen dosing in the ED preference lists? 2A1: Despite default values, ED providers may choose different initial doses of ibuprofen. So, it might be possible to retrospectively assess the delta for pain scores retrospectively across different initial doses of ibuprofen, if steps are taken to match patient characteristics and diagnoses.
2Q2: What steps should the residents take before conducting this research? 2A2: The residents should demonstrate an understanding of research ethics. They should submit an internal review board application stating the scope of the research and describing the need for their data and their plans to access and secure it. Finally, they should demonstrate proficiency in safely and efficiently running queries using the self-service tools.
2Q3: Name several alternatives to providing the residents with data warehouse access. 2A3: An honest broker can run (or receive) the report for the relevant fields of interest, and de-identify it, giving the residents data they need for the research without risking PHI (protected health information) loss. Or a datamart could be developed, so the residents can't access data out of scope of their project. The residents could access a de-identified self-service query tool, if available.
3. A vendor claims they've developed an "AI" algorithm that is more sensitive and specific at identifying patients with sepsis in the ED. The algorithm depends on so many dynamic patient variables to function, it cannot be properly evaluated just by inspection. Determine what different characteristics those patients had with your hospitals' patients, and whether those are likely to be significant. -Partner with the vendor on researching and developing the sepsis algorithm, tailored to your ED patient population, through use of historical data in the data warehouse. -Sign a non-disclosure agreement with the vendor, build a secure interface between your institution and the vendor, and test their algorithm on a sandboxed version of your EHR. This EHR can be populated by "synthetic" patients, or de-identified reflections of patients, or if a business agreement is in place, real patients whose demographic data and workup decisions are mirrored from the production environment. These approaches can be time-consuming and expensive, so if the algorithm doesn't perform well, there will be sunk costs.
3Q2: What are some risks of using AI algorithms for clinical care? 3A2: The biggest risk comes from the algorithm's complexity masking bias. There are already many examples in healthcare where AI recommendations thought to be based entirely on clinical details were actually found to be based on race, socioeconomic status, or other factors. Because the recommendations often appear as a "black box" to clinicians, some time may pass before these biases can be detected and properly interpreted.
4. Your hospital wants to participate in the local HIE (health information exchange).
4Q1: As an ED physician, do you expect ED patients would benefit from health information exchange? 4A1: Access to HIE has been shown to reduce redundant testing in the ED and reduce costs, and many cases have been described where the HIE provided helpful data for reaching a diagnosis and treatment faster.
4Q2: What ED presentations are least likely to benefit from HIE? 4A2: HIE could potentially make a difference in any ED presentation, if there's a history of allergy or a potential for drug interactions that would otherwise be missed (particularly if patients are unable to share their allergies or medication lists). However, ED visits concerning minor traumas resulting in sprains, fractures, or lacerations seem less likely to benefit from the data in an HIE than, for example, a patient with recent surgery, or on chemotherapy, or managing complex chronic conditions. 4Q3: The hospitals participating in the HIE are on different EHRs. How will data about patients from different facilities be shared and accessible? 4A3: Most discrete data today -diagnoses, medications, vital signs, lab results, etc -are captured through standard terminologies (like ICD-10, RxNorm, LOINC, etc). Messages between facilities and HIE are also sent according to a common standard. Patients are matched with algorithms that permit some small degree of "fuzziness" so records from two facilities can be recognized as belonging to the same patient so long as there are very closes matches with name, birthday, address, and potentially other criteria. However, idiosyncrasies in how notes are categorized and described, and particular aspects of DIDACTICS AND HANDS-ON CURRICULUM Baker  each EHR, mean that browsing patient data supplied by an HIE is not as seamless as viewing a chart compiled within a single institution.
4Q4: For this new HIE, your hospital wants to require a separate consent for patient data, so that neighboring facilities must document their own consent from the patient to access your hospital's data. Is this feasible? Why would the hospital pursue this? How does this balance patient privacy vs patient care needs? 4A4: This arrangement is technically possible (attempts at downloading HIE data from your hospital's institution can be met with a specialized consent form) and may be viewed by the hospital as patientcentric (if they have data at the hospital they'd rather not disclose to outside parties). However, this extra consent introduces an additional step for obtaining access, making routine use of HIE data less likely in the ED, and possibly jeopardizing care if critical details are missed or overlooked (or more difficult to collect, such as if patient arrives unconscious at the other facility).